lmdeploy serve api_server Qwen/Qwen3-8B --backend pytorch --logprobs-mode raw_logprobs --enable-metrics

python profile_restful_api.py --backend lmdeploy --dataset-path /nvme1/shared/ShareGPT_V3_unfiltered_cleaned_split.json --dataset-name random --random-input-len 170 --random-output-len 2048 --random-range-ratio 0.9  --num-prompts 1024 

============ Serving Benchmark Result ============
Backend:                                 lmdeploy  
Traffic request rate:                    inf       
Successful requests:                     1024      
Benchmark duration (s):                  319.86    
Total input tokens:                      165130    
Total generated tokens:                  1992686   
Total generated tokens (retokenized):    0         
Request throughput (req/s):              3.20      
Input token throughput (tok/s):          516.26    
Output token throughput (tok/s):         6229.89   
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   214800.21 
Median E2E Latency (ms):                 220168.95 
---------------Time to First Token----------------
Mean TTFT (ms):                          2856.35   
Median TTFT (ms):                        2831.78   
P99 TTFT (ms):                           4512.24   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          109.07    
Median TPOT (ms):                        112.41    
P99 TPOT (ms):                           165.81    
---------------Inter-token Latency----------------
Mean ITL (ms):                           942.11    
Median ITL (ms):                         380.80    
P99 ITL (ms):                            1191.39   
==================================================

============ Serving Benchmark Result ============
Backend:                                 lmdeploy  
Traffic request rate:                    inf       
Successful requests:                     1024      
Benchmark duration (s):                  305.74    
Total input tokens:                      165130    
Total generated tokens:                  1992686   
Total generated tokens (retokenized):    0         
Request throughput (req/s):              3.35      
Input token throughput (tok/s):          540.10    
Output token throughput (tok/s):         6517.59   
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   204367.81 
Median E2E Latency (ms):                 209678.84 
---------------Time to First Token----------------
Mean TTFT (ms):                          2798.76   
Median TTFT (ms):                        2643.68   
P99 TTFT (ms):                           4446.10   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          103.68    
Median TPOT (ms):                        106.52    
P99 TPOT (ms):                           158.08    
---------------Inter-token Latency----------------
Mean ITL (ms):                           560.02    
Median ITL (ms):                         225.71    
P99 ITL (ms):                            766.30    
==================================================